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ABS't’BACT 

In an experiment to test the effectiveness of 
discrimination training--contra sting good and poor teacher behaviors 
and demonstrating the stimulus occasions for these behaviors--as 
compared with feedback from axi instructor in microteaching, a 
discrimination model for a limited class of teaching behaviors was 
devised and subjects were divided into three groups according to the 
type of training received. The first group received discrimination 
training, practice, and feedback (DPF) ; the second group received 
discrimination training and practice (DP); and the third group 
received practice and feedback (PF) . A comparison of pre- and 
posttest scores on several behavior rating scales showed consistent 
and significant improvement of th^^ DPF and DP groups as compared with 
slight improvement in the PF group. These results seem to indicate 
that discrimination training could replace feedback from an 
instructor in many cases and thus effect substantial economies in 

(An appendix contains pretest posttest comparison 



t eacher 
g ra ph s) 



educat ion, 
(RT^) 



iiiMiiliiWiiMii 



EDO 39175 




m 



mm 



U.S. OEPARTMEHT OF HEALTH, EDUCATIOH & WELFARE 
nFFirF OF EDUCATIOH 




THIS OOCUHEKT HAS BtEK BEPMOUCEI) !S^^nPIMS 

PEBSOH 0« OBOAHIZATIOH 0BI0IHAIIH6 IT. ™ 

STATED DO HOT HECESSARILY REPRESEHT OFFICIAL OFFICE OF EDUCATIOH 

POSITIOH OR POLICY. 



. I 

Discrimination Trainin'^ and Feedback in Rhanin^ Teacher Behavior 
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Lauren B. Resnick & Lois K. Kiss 
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In learnins!: a comnlex social skill such as teaching? an iniivid,ual 
must acnuire both a new renertoire of behaviors and a set of discrimina- 
tions a,s to when each specific behavior in that renertoire is anpronriate. 
Studies in ”microteG,cliin.«?: ’ have shown that snecified teaching behaviors 
can be effectively shared or modix'ied through a I'lrocess of repeated prac- 
tice interspersed with critical review by the trainee of his own video- 
taped performance (Allen and Rvan, 1969). A study bv McDonald & Allen 

"tl*‘ fi t 

( 1967 , Chapter IV) showed /provision of ’’models” of effective performance 
of the teachin/ 3 ? behaviors to be acauired enhances acauisition of those 
behaviors to some deforce. However, .the model's effectiveness was eireatest 
when S viewed it alonff with E, who pointed out both the appropriate behaviors 
of the model and the occasions (student responses) suitable for those be- 



haviors . 

McDonald and Allen (196T» Chanter II ) also found that S^s who had feed- 
back sessions in which E helped them review videotapes of their performance 
showed significantly (greater improvement than ^who viewed their videotapes 
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alone. This effect of feedback from a supervisor remained even when 
pre-’»ractice model .'3 were provided (McDonald and Allen, 19oT , Chapter 
IV). In both the modelin," ana the feedback contexts, E's commentary 
apparently functioned as a form of informal "'discrimination training," 
contrasting good and poor teacher behaviors and cueing the stimulus 
occasions for these behaviors. 

The present study explores the effect of a special form of 'dis- 
criminative modeling, ‘ in vdrich a succession of model performances are 
deliberately chosen to contrast appropriate and inappropriate teacher 
behaviors and stimulus occasions. It is designed to test the hypothesis 
that when both responses ana stimulus occasions have been adequately 
discriminated in advance of practice, Ss can apply the discriminations to 
their own subsequent behavior and thus learn to '’seif-edit” their o\rr\ 
performance. This ca.pahilltv should make it possible for '^s to benefit 
from practice \^ithout need of direct feedback from a supervisor or 
experimenter. 

Bandura, Boss and Ross (1963) have shown that when a variety of 
modei.s of a clsss of behavior are shown, children can acquire the class^ 
of behavior without the idiosyncratic features of any sincrle model’s 
Performance. P>s in effect learn to produce now exemplars of th^ behavior 
class through ohsorvipT multiple models of the behoviop. A later study 
(Bandura and ^Tarris, loff,) showed that children could learn a new class 
of grammatical mourns most effectively wh.on models of the sentence form 
were interspersed with sentences of other types and the child's attention 
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directed to aifferences •between "correct'' and "incorrect" sentenceB. 

It is this use of contrastin^z , multiple models thet we term "discrimina- 
tive modeliniz," and that we seeK to apnlc^ to adult’s learning a com- 
plex interactive reaertorire. 

Method^ 

For mrooses of tho study, a limited cXnss of toaohinc! Tiehavlors 
was examined. T^ese are all behaviors required in effective testlnr 
or dia»:nosis of ttie intellectual skills of vouna children, where test- 
ing is conducted i,n a one-to-ore "tutorial" tvne setting. The snecifio 
behaviors studied were chosen on the basis of detailed behaviors] analyses 

of tutorial teaching skills (Resnick and Kiss, IdTO). 

Three elements of training were identified: l) Discrimination Train- 

ing (Variable D); 2) Practice (Variable P): and 3) Feedback (Variable F). 

Discrimination training (Variable D) consisted of viewing, rating and 
discussing videotaned tutorial diagnostic sessions. Mscriminatlon ta.nes 
were constructed of "model sessions" interspersed with '-prohlem sessions," 
in order to contrast annronriate and inannronriate behaviors hy a teacher . 
Prohlem sessions were chosen to illustrate behaviors which caused lengthy 

or inaccurate testlnn:. 

For Practice sessions (Variable P) Ss came Individually or in pairs 
to a snecial laboratory. During practice sessions E?_s (teacher trainees) 
diagnosed several children on each of several related tasks. Each interaction 
was recorded on videotape, and after completing her ratings for a child ,S 
could view the tape before beginning work with another ohiln. 
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Feedback (Variable F) refers to feedback from the experimenter. 

In the two p^rouns receivin,?: feedback E served as evaluator, ratin" 
the trainee’s ner^^ormance and discussin^r the video nlayback during the 
nractice sessions. Practice sessions for these <^rouns were scheduled 
individually. 'Ihs: ;^roup not receivin^^ feedback practiced in nairs, 
alternatin/:; as tester and observer. They were free to view videotapes 
together and discuss each other's performance.^ The experimenter, how- 
ever, did not interact with ti^em.^ 

Varyins; the three elements of training produced three experimental 
fi^roups . One ^yroup (labelled D^F) received all three trainin'^; ei.em'^^nts, 
Discrimination, Practice a*^d ‘f’^eedback for experimenter. A second 
/5:roup (PF) had Practice and Feeiback, but no Discrimination trainin'^. 

The third aroup (DP) hnd Discrimination Train, in‘T and »>'act''ce, but no 
Fetidliack . 

(FLTDE l) Thi.s sli^e summarizes the entire experimental procedure 
for each of the three «j;rouPS. All aroui^s besran vdth an orientation 
session, in which they uere fa.miliarized with th---' purpose of the traininin; 
program, tho criterin that would be used in .ludyinr; their performance, and 
the materials they wou.Td be unin?:: in working with the chil-'ren. All yroups 
were then protested. !-'or the pretests, each trainee diafmosed the performance 
of five or six children on two tasks. The first task was a "multiple dis- 



P, 



"Limits on time made it necessary for Ps in this treatment to share 
t-'ie number of children avails, ble for testing durinr; a. Riven practice 
session. Thus, these Fs each tested only 3 children rather than the 6 
tested by Ss in Feedback treatments. 

3 

• Informal observations suRsest that trainees workins in pairs actually 
encased in ver'^r little discussion of each other's performance. 
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crimination," which called for the child to noint out nine basic shanes 
when each was named. The second task was a 'two-nart discrimination" 
and called for the child to distins;uish the numerals '"6" and "9". Ss 
later received discrimination and nractice in dia«rnosin.^ th^- two class_es 
of tasks, multiple discriminations and two-part discriminations, but did not 
practice the particular tasks used in the pre- and post-test sessions. 

The chi3.dren tested were U-6 years old, dravm from Headstart ana 
Kindergarten classes at a nearby elementary school. A remote-controlled 
camera recorded the pre- and post-test interactions for Inter scorin'^, 
purposes, but the videotapes were not shown to trainees. 

A p;roup discrimination training?: session came next for Bs in fijroups 
DPF and DP. The training; tape for this session focused on multiple 
discrimination tasks. Bs in all pjroups then had a practice session in 
which they tested children on multiple discrimination tasks. Two of 
the groups (DPF and PF) had experimenter feedback durina the practice 
session. The cycle of discrimination training and practice was then 
repeated, this time focusing on the testing of two-part dir criminations. 
Finally, all Ss were post-tested in an exact replication of the pre-test 
procedure. 

Re sult s 

Pre- and post-tests were scored on two gesieral measures of diagnostic 
competence and on a series of specific trainee behaviors. 

(slide 2) The first general measure was a Certainty rating, designed 
to assess the confidence with which independent observers could ,^udge the 
child’s ability to perform each of the tasks tested. Using the form show- 
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here ar.M workinf;: fro’^ vi-leotane rl.'iybacks , two inde-nendent 
tallied the child's resDonses to each item in the test and 
their certainty that a cr.ild could or could not ne-’form th'- 



observers 



then rated 
ohd active . 



A ratinr: of 3+ or 3~ indiCwated a hiyh certainty; 1- or 1+ or 0 a very 
low certainty. In avera/njinr; certainty ratings, si^ns were i^>;nored. 

This resulted in a mcasiire that "Tiives enual weialj'- to str'"'nr certainty 
that the child can rerform the task and strona certaint?/" that he cannot, 
For each trainee separate scores were computcvl for multiple liscrimina- 



tion tasks and for two-nart discrimination tasks, avera^rinrr across the 



children she tested in a nre- or nost-test session. 

(SLIDF 3)These graphs shovr the changes in nre-- and nost-t-^st mea’^s 
for the three experimental grouns on the Certainty men sure. The data 
for this and al.l suhf;ecuent measures were anal^^’zed using mul‘^-1.nl« covariance 
analysis (Cooley h Lohnes, I 07 O), a nrocedure whic} comnares post-test means 
which have been adiusted for pre-test level, and which also permits simul- 
taneous examination of sets of related measures in order to detect common 
trends. This capacity permitted the examination o^' scores for multiple snd 
two-part discriminations combined,, as well as of scores for each type of task 
separately. For this measure, there were no significant differences between 
groups, although differences were in the expected directions. 

(SLIDE The same finding is repeated for the second general measure, 
the average number of minutes s^ent with a child on a single task. All 
groups showed substantial reductions in the time required to complete a 
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diagnosis; and all firrouns also showed a sifijnif leant reduction in 
variability from ore- to nost-test. However, thor*^ was no differential 
effect for the three o^rouns. Thus, /siener^l dias:nostic skill was annarently 
not differentially affected by the three traininc; conditions used in this 
experiment . 

The next five measures are all based on direct observa.tlon of snecific 
component behaviors that were treated in the discrimination training tapes. 
Inter-observer reliability for these measures rano;ed from 80 to 100 percent. 

(slide 5) Durin/i: the pretest, many trainees tended to eliminate some 
of the choice objects durinfi; the testing? session, so that a child was choos- 
ing; his response from a e:roup of three or four rather than of nine or eleven 
objects. Often trainees eliminated some of the choices before retest inn;, 
raakinc; the retest ineffective because of the increased chance of a random 
response being correct. The slide shows changes from pre- to post-test in 
the behavior of eliminating choices. Differences between the groups were 

significant for the two tasks combined (F = 9.76)df =2,23; p<.0l) and for 

(F=T.26;df=2,?3;p<r.0l) 

the multiple discrimination task alone./ As you can see, on the multiple 
discrimination task the behavior was reduced nearly to zero in the two 
groups that haa niscriraination training, while the group without discrimina- 
tion training actually increased its rate of elimination behavior. 

(SLIDE 6) This is a measure of the number of times the trainee permitted 
the child to elirrinate choice stimuli by picking up or putting aside each 
object as he dea]t with it. Differences among treatment groups were significant 
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for the two tasks comhined (F=’3.44;df-2,23jP < •05) • The behavior 
occurred mainly in the multiple discrimination task. Although the 
large pre-test differences between the groups require the,t the post- 
test effects be interpreted with caution, the dif^'erences are in the 
direction nredicted for behaviors dealt with in discrimination training?. 
Occurences were reduced nearly to zero for the grouns with discrimination 
training (DP and DPP). Occurrence of the behavior actuallv increased on 
the post-test for the groun without discrimination training. This increase 
for the DP groun illustrates the limitations of relying on feedback during 
pr ctice sessions as a ma,1or training techninue. If a trainee did not 
harper, to test a child who elimina.ted materials in training, she did not 
know to handle the situation when it arose in the nost-test. The groiin 
that considered the situation in discrimina.tion training, however, knew 
how to deal with it even if it had not arisen in nractice sessions. 

(slide T) This is a measure of ’’cueing," defined as all physical 
indications a trainee made that might tend to influence a child’s choice 
in responding to a request. Behaviors scored as cueint^ included adding, 
pointing to, tanning, moving forward, or otherwise indicating a specific 
stimulus to the child ,iust before of after requesting a response. Differences 
between groups were significant for the two tasks combined (P=T.51;df=2,23; 
p<.0l) and for tbe multiple discrimination task alone (E“6.46;d.f=2,23 jp^.Cl) . 

The two groups undergoing discrimination training reduced the amount of cue- 
ing in their post-test performances nearlv to zero. The group which did not 
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experience discrimination traininn; (PF) was cueiiifr nearl 7 ei/?:ht times 
as often as tVie others on the multiple discrimination post-test, althous^h 
it had be^un with a slisihtly lower rate on the nretest, 

(:iLIi)E o) 'I’his is a measure of "■nromptinq;" I'ohavior, defined as 
any verbal information the trainee f^ave the child which miffiht direct him 
to choose the ri^ht response or the wrone; response, or to chans^e responses. 

A ’’prompt" was scored whenever a trainee named the stimulus oh^ects for 
the child, or p;avG him information about the ob.iect she wanted him to 
choose (e.fz;. "Show me a triano:le; it’s like a roof"); or when the trainee 
Rsked for an ob.lect the child was holding or touching, asked for a niece 
no lon!7:er on the table, or asked for numbers in order; or when the trainee 
nrovided feedback on incorrect ' resnonses h'r r,B,yinr. "no" or "try a-min." 

Group differences are sicinif leant for the tv/o tasks combined (F='4,UU;df=2,23; 
P<C»05). Oomnarison of the slones in the rranhs suan;est that for each task 
the nroxxn without discrimination training? (7'F) imnroved somewhat less than 
the other two. ‘•'ovrever, the hiii;her nromptin?? rate of rroup on the pre- 
test, esnecially in the multinle discriminp.tion, renuires that these results 
be interpreted with caution. 

(slide 9) Determininf? a child's ability to distinwish between two 
related stimuli renuires some elements in the test which are used to distract 
the child from the two relevant responses and force him to make a ne” 
resnonse each time he is asked rather than follcwinr a "back and forth” 
pattern. Two different measures of distract ins; behavior by the trainee are 
represented here. They were used only on two-part tasks. For the tor 
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raeasurp* Failin/2; to Acid Distractinf^ Stiiriuli, the observer checked each 
time the trainee used only the two relevant stimuli (i.e., ”6’' and ’’9'') 
durinp: some part of the task. For the bottom measure, Failin^^ to Distract 
Child between Responses, the observer checked each time the trainee re- 
tested both items without distractinf' the child between cycles either by 
asking for an irrelevant response or by shuffling the position of the 
materials. Thus, these are measures of a trainee's failure to distract 
annronr iately . 

Differences between the treatment (^rouns are significant for the two 
measures considered simultaneously (F^5*l^ »df=2 ,?? ; p^.05). Compsrison 
of slopes shows that the ^roun without discrimination traininj’i (PF) 
iranroved less than the other two (DPF and DP). For the ton measure. Fail- 
ing to Add Distracting Stimuli, however, the PF group began with a near 
zero occurence and thus had little room for imnrovement . Therefore, the 
differences in slone are nrobably attributable to the wide pre-test 



differences, rather than to post-test differences. Pre-test differences 
are less marked for the bottom measure. Failing to Distract Child, and 
post-test differences are sharper. While tne PF group exhibits the 

behavior on only 12 percent of tasks on the post-test, even such a 
relatively low occurence could seriously affect the accuracy of testing. 
The two groups that received discrimination training both completely 
eliminated the Failure to Distract behavior. 

Each of the behaviors just discussed had been specifically treated 
in the Discrimination training tapes. In each case, the two groups that 
saw the tapes (DPF and DP) improved significantly more than the group thr.\t 
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dir) not see the tanes , while feedback from the ex-perimenter offered 
no ad\rantafii:e . This findin<r sunrorts an interr>r»=‘tat1,on of discrimination 
■pre-trainin^' as esta.>'lishin<T conditions which mahe it ■possible for the 
trainee to '’edit'' his own behavior, thus decreasins* reliance on iud clients 
by others, 

Since there was no comiDarable effect of discrimination training? on the 
ficeneral measures of Certainty and Time, th^ effect annears to be specific 
to the behaviors focussed on in nre-trnirn* . For other behaviors, those 
for which the " self-edit i nr?:’' skill would not have been developed, it would 
be expected that expert feedback would be essential to improvement. One 
of the behaviors measured in the pre and post-tests allows us to test this 
inference, since there was no explicit attention to it in the discrimination 
training tapes, 

(SLIDF 10) In this measure, occasions on which a retest should have 
been made but was not were counted. Occasions renuirinc; retestinif?: are those 
in which the response is prompted or cued or the response mlR;ht be correct 
by chance. The discrimination trainin<y tape did not focus on the retestincr 
skill explicitly althouf^h it was touched UPon tan^rentially , The measure was 
derived from the observer’s certainty ratin.v forms, A count was made of 
the items rated less than ”3” by one or more observers where no retest was 
performed by the trainee. 

Differences between the experimental «crouPs were significant for the 
two measures combined (F=:*+.t)^;df=2,23"p c.O^) for the two-nart discrimina- 
tion considered senaratelv (Fs:3-TT;df=2j23 ;pc,0? ). In this case, the a:rou.p 
without feedback showed no improvement on either type of task, while the 
other two prroups showed eouivalent reductions in means (takinnr pretest 
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performance into account). Thus, where nre-traininc? has not explicitly 
developed the self-’editins!: capacitv, feedback appears to be the critical 
variable in improving performance. 

DISCUSSION 

The results of this experiment support the view that discriminative 
modeling; techniques can produce a capacity for "self-editing” which 
substantially reduces a trainee’s reliance on outside feedback. The 
study demonstrates that modeling techniques can be effectively used in 
developing; interactive skills in adults. In practical terms, di’scrimina- 
tion pretrainin," offers the possibility of doing; away with the costlv 
one-to-one feedback that has been found to be a critical requirement in 
other raicroteachina; experiments. Usints: multiple models on film or video- 
tape, discrimination sessions could be provided for c;roups at a central 
location; trainees could then practice in times and places convenient to 
them, 

Althoug;h promising, the technique of discriminative pretrainingr 
requires further investigation. Most important, it is necessary to 
determine, whether effects on specific component behaviors can be cumulated 
to produce a difference in general nerfoi’raance. Further experiments are' 
planned to test directlv the hypothesis that if enough specific components 
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are acnui?.'ed and or 'rani zed into a STnooth T)erformance, /renaral measures 
will also res:ister an imnrov'=*Tnent , If nxic'^ an imrrovement can be 



effected throu?r?i the use of discriminative modeT Inr without direct 
feedback, substantial economy and improved effectiveness in teacher 
train! efforts can be anticipated . 
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G roup 
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DPF (10 S’s) 


PF (10 S’s) 


DP (8 S’s) 


Ox'ientation 
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XXX 
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Pretest 
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XXX 
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Group Discrimination Training 
(Multiple Discriminations) 


XXX 




XXX 


Practice 

(Multiple Discriminations) 


XXX 

(6 children 
per trainee) 


XXX 

(6 children 
per trainee) 


XXX 

(3 children 
per trainee) 


Feedback 

(Multiple Discriminations) 


XXX 

(experimenter 

videotape) 


XXX 

& (experimenter & 
videotape) 


(peer & 
videotape) 


Group Discrimination Training 
(Two-part Discriminations) 
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Practice 

(Two-part Discriminations) 


XXX 

(6 children 
per trainee) 


XXX 

(6 children 
per trainee) 


XXX 

(3 children 
per trainee) 


Feedback 

(Two-part Discriminations) 


XXX 

(experimenter 

videotape) 


XXX 

■ & (experimenter & 
videotape) 


(peer & 
videotape) 


I Post-test 


XXX 


XXX 


XXX 



Summary of training and test sessions for the three experimental groups. 
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Numeral 



* 

Objective Given; An unordered array of numerals. 

Child can: Point to each numei‘al as it is named. 



Tally 



•M 

o 



cd 

o 

>> 

o 

• 4 -> 

•a 

tfH 

CD 

Q 



cj 

O 



u 



t 

o 



a 

CO 

CO 

o 

pH 



0) 



§ 

O 



d 

cd 

o 

P— 4 
•iH 

CO 

CO 

o 

pH 



s 

a 

I 

■§ 

■8 

pH 



§ 

>> 

0) 



o 

Q 



2 - 



1 - 



0 



1 + 



2 + 



3+ 



3- 



0 



1 + 



2 + 



3+ 



3- 



2 - 



1 - 



0 



1 + 



2 + 



3+ 



3- 



2 - 



X- 



0 



l-H 



2 + 



3+ 



3- 



0 



1 + 



2 + 



3+ 



/Tt 

u 



3” 



1 - 



0 



l-i- 



2 + 



3+ 



V 



3- 



0 



1 + 



2 + 



3+ 



8 






0 



1 + 



2 + 



3+ 



2 - 



1 - 



0 



1 + 



2 + 



3+ 



Rating form for Certainty judgments 
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TWO-PART DISCRIMINATION TASK MULTIPLE DISCRIMINATION TASK 



DPF 







PF 




-A 


DP 


■— 


m 




D INDICATES DISCRIMINATION TRAINING; P INDICATES 
PRACTICE; F INDICATES EXPERIMENTER FEEDBACK 





PRE POST 



MEAN CERTAINTY RATING OF OBSERVERS ON PRETEST 
AND POSTTEST FOR EACH TREATMENT GROUP 
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TWO-PART DISCRIMINATION TASK MULTIPLE DISCRIMINATION TASK 
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TRAINING; P INDICATES 

PRACTICE; F INDICATES EXPERIMENTER FEEDBACK 




Mllfli! PRETEST AND POSTTEST FOR EACH TREAT- 

MENT GROUP 
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TWO-PART DISCRIMINATION TASK MULTIPLE DISCRIMINATION TASK 





D INDICATES DISCRIMINATION TRAINING; P INDICATES 
PRACTICE; F INDICATES EXPERIMENTER FEEDBACK 




MEAN PERCENT OF TASKS IN WHICH TRAINEE ELIMINATES 
PART OF STIMULI ON PRETEST AND POSTTEST SESSIONS FOR 
EACH TREATMENT GROUP 
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MULTIPLE DISCRIMINATION TASK 
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D INDICATES DISCRIMINATION TRAINING; P INDICATES 
PRACTICE; F INDICATES EXPERIMENTER FEEDBACK 
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MEAN PERCENT OF TASKS IN WHICH TRAINEE ALLOWS 
CHILD TO ELIMINATE PART OF STIMULI ON PRETEST 
AND POSTTEST FOR EACH TREATMENT GROUP 
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TWO-PART DISCRIMINATION TASK MULTIPLE DISCRIMINATION TASK 
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D INDICATES DISCRIMINATION TRAINING; P INDICATES 
PRACTICE; F INDICATES EXPERIMENTER FEEDBACK 




PRE POST 
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MEAN NUMBER OF C^UES PER TASK ON PRETEST AND POSTTEST 
FOR EACH TREATMENT GROUP 
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TWO-PART DISCRIMINATION TASK MULTIPLE DISCRIMINATION TASK 
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D INDICATES DISCRIMINATION TRAINING: P INDICATES 
PRACTICE; F INDICATES EXPERIMENTER FEEDBACK 




MEAN NUMBER OF VERBAL PROMPTS PER TASK ON 
PRETEST AND POSTTEST FOR EACH TREATMENT GROUP 
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TWO-PART DISCRIMINATION TASK TWO - PART DISCRIMINATION TASK 
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MEAN PERCENT OF TASKS IN WHICH TRAINEE FAILS TO ADD 
DISTRACTING MATERIALS BY PRETEST AND POSTTEST FOR 
EACH TREATMENT GROUP. 




MEAN PERCENT OF TASKS IN WHICH TRAINEE FAILS TO 
DISTRACT CHILD BETWEEN RETESTS ON PRETEST AND 
POSTTEST FOR EACH TREATMENT GROUP 
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TWO-PART DiSCRSMlNATION TASK MULTIPLE DISCRIMINATION TASK 








D INDICATES DISCRIMINATION TRAINING; P INDICATES 
PRACTICE; F INDICATES EXPERIMENTER FEEDBACK 
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MEAN ITEMS NOT RETESTED MARKED UNCERTAIN BY 
OBSERVERS ON PRETEST AND POSTTEST FOR EACH 
TREATMENT GROUP 
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